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Abstract 


This article describes the development of the Community-Based 
Learning (CBL) Scorecard by a grant-funded consortium of lib¬ 
eral arts institutions. The aim of the scorecard was to promote 
assessment that improves student learning with an instrument 
that employs a quantitative scale, allowing for benchmarking 
across institutions. Extensive interviews with faculty who par¬ 
ticipated in the pilot (Charles & Choi, 2013), including specific 
perceptions of the value of using the scorecard to assess the stu¬ 
dents’ CBL experience and improve learning outcomes, were 
reviewed. Results indicated that the CBL Scorecard, with appro¬ 
priate administration, serves as an adaptable tool for assessment 
of CBL that can provide timely feedback and reminders of best 
practices to faculty. Increasing student response rates, the score¬ 
card’s value for faculty development, and improving the experi¬ 
ence for community partners are discussed. 


Introduction 



mportant conversations are occurring locally and nationally 


about the value of higher education, including the impact of 


community engagement on college students and the vital role 
community partners play in the coeducation of students. Ideally, 
initiatives designed to address community opportunities and con¬ 
cerns dovetail with frameworks that support student learning. This 
article focuses on community-based learning (encompassing for 
our purposes service-learning and community-based research), 
generally acknowledged to be complex and multidimensional 
(Eyler & Giles, 1999; Mackaway, Winchester-Seeto, Coulson, & Harvey, 
2011; McDonald, 2012; Zhang et al, 2011 ). Community-based learning 
has pedagogical aims, but those aims are joined to those of social 
justice and, more specifically, to meeting the needs of the commu¬ 
nity. The tasks and responsibilities of the students are aligned with 
course content, but must also work within the structures of the 
local partnership. As a form of experiential education, community- 
based learning may be utterly unpredictable as it unfolds, messy as 
a day-to-day experience, and shaped by serendipitous encounters 
with individuals from backgrounds and realms of experience that 
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may differ greatly from those of traditional-aged college students. 
Instructors who employ community-based learning in their courses 
year after year recognize intuitively the complexity and depth of 
learning that takes place, and that it is qualitatively different from 
what students glean from textbooks and classroom lectures. But 
measuring this learning, capturing its difference from traditional 
learning in institutions of higher education, is not simple. 

One challenge is capturing what maybe a qualitative difference 
using quantitative measures. Researchers in the growing assess¬ 
ment of learning movement often face difficulties in navigating the 
divide between quantitative and qualitative approaches, and such 
difficulties may reflect another tension: that between assessment 
for accountability and assessment for improvement, as Peter T. 
Ewell frames it. According to Ewell (2009), assessment for improve¬ 
ment typically relies on both qualitative and quantitative evidence, 
whereas assessment for accountability requires demonstration 
of “conformity with an established standard of process or out¬ 
come” and hence quantitative, cross-institutional measures (p. 8). 
Although most assessment specialists see improvement of student 
learning as the foremost goal, many recognize that such results are 
still unfortunately rare or, at least, difficult to demonstrate (Angelo, 
1999; Banta & Blaich, 2011; Blaich & Wise, 2011). 

Research Goals and Questions 

This article reflects on the development of the Community- 
Based Learning (CBL) Scorecard by a consortium of liberal arts 
institutions, funded by grants from the Teagle Foundation (here¬ 
after referred to as the Teagle Consortium). The consortium chose 
to use the term “community-based learning” as “accurately descrip¬ 
tive” of the programs and courses to be assessed and “unburdened 
by the negative connotations and history implied by a client- 
server’ dynamic between the community and colleges engaged in 
CBL” (Rhodes College, Systematic Assessment of Student Learning, 2008, 
p. 8). The aim in creating the scorecard was to promote assess¬ 
ment that improves student learning, but with an instrument that 
employs a quantitative scale, allowing for benchmarking across 
institutions. The CBL Scorecard was developed both to assess “the 
value added of CBL programming on student learning and civic 
engagement” and to “close the assessment loop by developing a 
process for applying Scorecard results to course/program improve¬ 
ment and by broadly disseminating and encouraging the use of the 
protocol and collected data institutionally, regionally and nation¬ 
ally” (Rhodes College, Systematic Assessment of Student Learning, 2008, 
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p. 1 ). Our aim is to assess the assessment: to clarify ways in which 
the CBL Scorecard has succeeded, as well as to acknowledge its 
limitations. For such analysis, we rely on the results of research 
carried out by Drs. Robiaun Charles and YuKang Choi as doctoral 
students at Vanderbilt University’s Peabody College of Education 
and Human Development in the form of extensive interviews with 
Teagle Consortium faculty who participated in the CBL Scorecard 
pilot and on our own experience with the scorecard at our home 
institution, Hobart & William Smith Colleges, supplemented by 
comments provided to us by participating institutions. 

Our analysis is framed within questions raised in the literature 
on assessment of service-learning and community-based research 
as well as the current literature on assessment of student learning 
outcomes in general, such as how to implement assessment in such 
a way as to facilitate improvement of learning, or how to develop 
models that are both sustainable and transformative. Using these 
research-based recommendations for the employment of the CBL 
Scorecard should provide an adaptable tool for guiding the change 
that leads to improved student learning outcomes in CBL courses 
and programs. 

A Different Epistemology 

A starting point in considering how to assess community- 
based learning is the recognition that not only is there a distinct 
knowledge base buttressing this practice (Brammer et al, 2012; 
Shapiro, 2012), but also that its epistemological bearings are distinc¬ 
tive (Butin, 2010; Eyler, 2009; Eyler & Giles, 1999). As Janet Eyler (2009) 
observes, “Knowledge in the classroom tends to be compartmen¬ 
talized into disciplines, whereas in use in the community or work¬ 
place it tends to be organized around problems or domains of prac¬ 
tice” (p. 29). The student experience of such knowledge is unique, 
as the impact of actual community engagement is both broad and 
deep. Thus community-based learning adds an affective compo¬ 
nent to the acquisition of academic content, complicating and 
developing cognition with real-life application of concepts learned. 
Tellingly, assessment of the learning in community-based learning 
has taken diverse directions. Assessment instruments and methods 
have been devised to measure not only cognitive outcomes, but also 
changes in civic engagement, personal development, intercultural 
understanding, organizational and communication skills, and most 
recently empathy (Bringle, Clayton, & Hatcher, 2013; Everhart, 2016; 
Eyler & Giles, 1999; Ming, Lee, & Ka, 2009). 
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Although community-based learning is a cross-disciplinary 
pedagogy, it also supports discipline-based learning and knowledge 
production—in economics, sociology, political science, environ¬ 
mental science, history, rhetoric, architectural studies, education, 
and religious studies, to name only a few. At the same time, with 
an experiential base it is “holistic in nature” (Mackaway et al, 2011, p. 
1 ). Hence the challenge of charting the difference that community- 
based learning makes, given that learning outcomes may reflect 
many fields, realms of experience, and facets of individual growth. 

Assessment as Research, Assessment Based 
in Research 

The limitations of some approaches to assessment of commu¬ 
nity-based learning have been studied. A common approach is to 
survey students at the completion of a community-based learning 
experience, but when such a survey asks students to report on 
their learning as opposed to demonstrating their learning, satis¬ 
faction may be indistinguishable from actual learning. (The in- 
house survey we employed at Hobart & William Smith Colleges 
before the development of the CBL Scorecard suffered from this 
limitation.) Such reports are “a weak measure of the complex cog¬ 
nitive outcomes we expect from service-learning” (Eyler, 2000, p. 
13). Ideally such indirect assessment would be paired with a more 
direct assessment of learning if the aim is measuring academic and/ 
or cognitive learning outcomes, as opposed to a sense of personal 
growth or commitment to community engagement. Assessing 
embedded student work is one viable and frequently proposed 
alternative or complementary assessment method (Fitch, Steinke, & 
Hudson, 2013; Molee, Henry, Sessa, & McKinney-Prupis, 2010; Shapiro, 
2012 ). Yet developing criteria and rubrics that may be employed 
across multiple disciplines when aspects of the learning are disci¬ 
pline- and/or course-specific is challenging, to say the least. 

A related approach that may be less discipline-specific focuses 
on student reflection in the form of writing and/or discussion, a 
component of community-based learning critical to its effective¬ 
ness and impact (Eyler & Giles, 1999). Yet questions arise here, too: 
Students vary in their reflectiveness; writing skills impact the 
quality and clarity of reflections; some students for personal or cul¬ 
tural reasons experience discomfort with personal revelation that 
may be a part of reflection, and respond instead with superficial 
or dissembling entries; reflection may be more common in some 
disciplines than in others; and the degree to which the instructor 
communicates, designs, and structures the reflection exercise for 
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students matters (Mackaway et al, 2011, pp. 7-8). Nonetheless, such 
direct measures and others, such as evaluating student problem¬ 
solving competency through interviews (Eyler & Giles, 1999), may 
be effectively employed and may compose part of ongoing research 
on how community-based learning supports the goals of a liberal 
education (Eyler, 2000; Jameson, Clayton, &Ash, 2013). 

Such studies raise the question of the relationship between 
assessment and research. Undoubtedly after 35 years of research 
since service-learning gained recognition as a field (Eyler & Giles, 
2013), any assessment ought to rely on research, no matter which 
facet of the learning is under consideration. Further, the relation¬ 
ship between research and assessment is reciprocal: assessment is 
as important to ongoing research as research is to assessment. Eyler 
and Giles observe: “The assessment of the effective implementation 
of key elements of service-learning are critical for strengthening 
research in this field” (p. 55). In sum, research into the learning in 
community-based learning gains when an assessment method or 
instrument is research-based, focused on defined outcomes, and 
rolled out with ongoing inquiry into its strengths and limitations. 

TheTeagle Consortium’s Community-Based 
Learning (CBL) Scorecard 

The initiative to create a replicable instrument to assess com¬ 
munity-based learning began with a collaboration among faculty 
and administrators at Rhodes College, Niagara University, and 
Franklin & Marshall College with a Teagle Foundation planning 
grant to fund the initial stages (2007-2008), followed by the award 
of an implementation grant. 

The goals for both planning and implementation were as 
follows: 

1. Systematically assess the value added of CBL program¬ 
ming on student learning and civic engagement, using 
the CBL Scorecard we developed for measuring CBL 
course/program effectiveness; 

2. Close the assessment loop by developing a process 
for applying Scorecard results to course/program 
improvement and by broadly disseminating and 
encouraging the use of the protocol and collected data 
institutionally, regionally, and nationally and; 

3. Expand and sustain a consortium of liberal arts col¬ 
leges committed to establishing and sharing effective 
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practices for the assessment of community-based 
learning. Two sub-goals for the consortium are to: 

a. Disseminate information about the impact of 
CBL on student cognitive learning and; 

b. Create a culture of assessment on the cam¬ 
puses of participating institutions (Rhodes 
College, Systematic Assessment of Student Learning, 
2008, p. 1). 

Among the activities undertaken during the planning phase were 
reviews of existing assessment instruments for community-based 
learning and of the literature about such assessment. One result 
of this review process was the finding that “several large research 
studies have already established the connection between effec¬ 
tive practices for community-based learning courses/programs 
and student learning outcomes.” Consortium members decided 
“to build on this research, rather than duplicate it, using success 
factors linked by research to student learning as the basis for our 
Scorecard” (Rhodes College, Systematic Assessment of Student Learning, 
2008, p. 2). This choice was key as consortium members confronted 
the challenge of developing an instrument that was replicable 
and could be used across disciplines, programs, and institutions, 
thereby affording cross-institutional benchmarking. The focus of 
the instrument would be practices in community-based learning 
courses and programs found to enhance cognitive learning out¬ 
comes as opposed to affective ones. Research that was especially 
germane to the identification of these effective practices or “success 
factors” included the work of Astin, Vogelgesang, Ikeda, and Yee 
(2000); Eyler and Giles (1999); and Eyler, Giles, Stenson, and Gray 
(2001). In selecting success factors, consortium members chose 
those appropriate to a cross-disciplinary instrument, eliminating 
such variables as “the student’s degree of interest in the subject 
matter,. .. since CBL programs/courses at liberal arts institutions 
are not necessarily tied to a specific major or interest” (Rhodes 
College, Systematic Assessment of Student Learning, 2008, p. 8). 

The Teagle Consortium recruited new members for a total of 
11 liberal arts colleges and universities by 2011, which enlarged the 
pool of courses and programs for the pilot of the CBL Scorecard. 
This increase made possible collaboration among a larger group of 
practitioners as the scorecard was further developed, revised, and 
refined, and helped create a community of practice. We at Hobart 
& William Smith Colleges were deeply grateful for the outreach of 
Niagara University. As one of 28 baccalaureate colleges earning the 
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2010 Carnegie engaged designation, we value enhanced assessment 
of community-based learning at the course and program level for 
the ongoing improvement of learning, and we were excited to join 
in this promising collaboration with peers from similar institutions. 

After an initial pilot of the CBL Scorecard at Franklin & 
Marshall, Niagara, and Rhodes during the planning phase of the 
project, the consortium worked with Dr. John Braxton (Vanderbilt 
University) and Dr. Willis Jones (then a graduate student at 
Vanderbilt, now faculty at University of Kentucky) to build a more 
robust data collection instrument (Rhodes College, Community- 
Based Learning, 2010). Braxton and Jones began by engaging in an 
extensive literature review of the research on effective pedagogies 
and practices for community-based learning. From this literature 
review a list of statements regarding best practices in community- 
based learning was obtained. This large pool of best practice state¬ 
ments was then reviewed by a group of educational practitioners 
and researchers for clarity and face validity. The reviewers selected 
32 of these best practice statements, which were grouped into four 
“domains of practice”: (1) placement quality, (2) application and 
connection to academic learning, (3) reflection, and (4) quality of 
community partnerships. 

To further explore the validity and relative importance of the 
32 statements, Braxton and Jones recruited nationally recognized 
experts in community-based learning to complete a Q-sort task. A 
Q-sort is a comparative process in which respondents are required 
to sort their responses to statements into a predetermined number 
of piles. For this analysis, respondents were asked to place the 32 
best practice statements into five piles, assigning rankings ranging 
from “not essential at all” to “most essential” for high-quality 
community-based learning. In Q-sort tasks, response categories 
are generally forced into an approximately normal distribution by 
specifying the number of statements that can be placed in each 
pile on the scale. For this Q-sort task, respondents were forced to 
place four statements in each of the two most extreme categories 
(“not essential at all” and “most essential”), six statements each 
were forced into the “somewhat essential” and “very essential” cat¬ 
egories, and 12 statements were forced into the “essential” category. 

Braxton and Jones mailed this Q-sort task to eight community- 
based learning experts and received a response rate of 50%. They 
then created weights for each best practice statement based on the 
findings of the Q-sort task. Each response pile in the Q-sort was 
given a value ranging from 1 (“not essential at all”) to 5 (“most 
essential”). Next, they calculated means scores for each ques- 
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tion based on the Q-sort results and used those means scores as 
response weights in the CBL Scorecard. Question means scores 
ranged from 1.25 to 4.25. Thus at multiple stages (i.e., review of 
literature in planning stage; Braxton and Jones review of literature; 
survey of top specialists), the development of the CBL Scorecard 
drew on the most up-to-date and widely recognized research in 
the field. Given the extensive literature reviews and testing via field 
experts, we are confident of the face and content validity of the CBL 
Scorecard. As the consortium notes in the Year 5 progress report to 
the Teagle Foundation, “This approach is unique in both its focus 
on the course or program as the unit of analysis and its foundation 
in existing research on effective service learning practice” (Rhodes 
College, Community-Based Learning, 2013, p. 4). 

Other crucial aspects of the CBL Scorecard were also unique 
and, we felt, important steps forward for assessment of commu¬ 
nity-based learning. Versions of the same instrument were devel¬ 
oped for three audiences: faculty, students, and community part¬ 
ners. The inclusion of community feedback was a central aim of 
the consortium from the start; surveying all three groups with ver¬ 
sions of the same instrument was a significant innovation. Further, 
although the instrument is a quantitative one, its focus on best 
practices as opposed to the measurement of particular learning 
outcomes avoids measuring only the most easily measured hall¬ 
marks of learning, a potentially reductionist analysis. It also avoids 
the murkiness of self-reporting of learning on which many other 
survey-based assessments rely, as it asks about the degree to which 
students experienced certain practices, not about their sense of 
success with learning. Finally, the CBL Scorecard includes only 
practices that have been found to be valuable to community-based 
learning across disciplines and programs, obviating the need to 
adjust the scorecard for different courses and discipline-specific 
subject matter. 

Assessing the Assessment: Research by 
Charles and Choi 

Having administered the CBL Scorecard over five semes¬ 
ters (spring 2011-spring 2013), the Teagle Consortium invited 
Robiaun Charles and YuKang Choi, doctoral candidates at 
Vanderbilt University’s Peabody College of Education and Human 
Development, to conduct interviews with faculty participants as 
part of a mixed methods research project. With “the goal of making 
the instrument more useful as a classroom diagnostic tool that can 
be readily used by instructors without professional interpretation,” 
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the consortium collaborated with Charles and Choi to generate the 
following research questions: 

1. What are the practices responsible for high performing 
community-based learning? 

2. What are recommendations that can make other com¬ 
munity-based learning courses better? 

3. What is an appropriate process/protocol for assessing 
community-based learning courses on an ongoing 
basis? (Rhodes College, Assessing Community-Based 
Learning, 2013, pp. 4-5) 

Charles and Choi used quantitative data gathered through the 
administration of the CBL Scorecard to identify high-performing 
courses, selecting 48 out of 90 courses in the consortium database, 
eliminating 42 because of incomplete responses. Within these 48 
courses, the researchers employed theoretical sampling to identify 
high-performing courses, thereby narrowing the sample to 30 for 
a qualitative study. They invited the instructors of these courses to 
be interviewed by phone, with the result that faculty teaching 21 
high-performing courses participated in the study, a sample that 
represented 70% of the faculty teaching high-performing courses 
and seven of the consortium institutions. (See Charles & Choi, 2013, 
for a full description of the methodology they used to construct 
the sample [pp. 16-21] and of the data analysis and coding of the 
transcripts [p. 24].) The interviews lasted 60-90 minutes. As Charles 
and Choi remark, “Although the scorecard provided information 
about what was happening in these selected courses, we conducted 
the interviews to learn how it was happening, why it was happening 
and when it was happening” (p. 22). 

For the purposes of this article, the third research question 
is most germane: “What is an appropriate process/protocol for 
assessing community-based learning courses on an ongoing basis?” 
Charles and Choi (2013) identifies six themes that arose through the 
interviews and their subsequent data analysis: 

1. Light bulb moments 

2. What exactly do the results mean? 

3. How are scorecard results going to be used? 

4. Be mindful of the community 

5. Administration and execution of the scorecard needs 
to be standardized 

6. A necessary evil (p. 47) 
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The “light bulb moments” experienced by faculty came mostly 
in the form of reminders of how to plan community-based learning 
courses intentionally rather than seeing the scorecard results as 
providing new information. Nonetheless, faculty comments as 
reported by Charles and Choi (2013) were overwhelmingly positive. 
Respondents found the scorecard helpful, remarking, for instance, 
that “it helped you think more comprehensively about what we 
wanted for our course” (p. 48); “you think more broadly about course 
design because you have a scorecard as a framework to operate out 
of” (p. 48); and “this is what you want to look for when you’re put¬ 
ting together your syllabus, when you’re creating your course or 
when you’re working with an institute or center or whomever you 
are working with in putting it together” (p. 49). Only one comment 
included in Charles and Choi’s report seemed to question the value 
of the scorecard (classified under “a necessary evil”): This faculty 
member wondered whether there is not “an inherent tautology in 
developing these measurement scales to prove what good outcomes 
are when they’re developed to prove what good outcomes are” and 
called for more longitudinal studies and other ways to measure the 
impact of community-based learning courses (p. 52). 

Faculty did identify limitations or weaknesses in the scorecard. 
An important flaw that was, at least partially, addressed in subse¬ 
quent administrations of the instrument was the delivery of the 
results. The earliest feedback came in the form of raw data; faculty 
asked instead for more easily interpretable results (Charles & Choi, 
2013). There was also almost universal concern that the data could 
be used for faculty evaluation rather than course development and 
improvement; if that were the case, faculty would be reluctant to 
participate. Further, as courses vary, faculty noted that it should be 
used as a guide; the individual instructor could then decide how 
it might best inform course design and practice (p. 50). Finally, the 
faculty interviews made evident a need to standardize the admin¬ 
istration and execution of the scorecard, because “ [m] any partici¬ 
pants also were not clear on the who, what, when, where and how 
of the scored execution and administration in their classes” (p. 51). 
Clearly it is vital for institutions to establish a protocol for admin¬ 
istration that is transparent and convenient for faculty and yields 
a high response rate. 

Another limitation of the scorecard articulated in these inter¬ 
views concerns the participation of community partners. Although 
the desire to include the community voice in assessment was a pri¬ 
ority for the CBL Scorecard project, in effect, community partners 
sometimes lacked access to computers. Perhaps more damning 
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than problems of delivery was that of perspective: Many of the 
scorecard questions were not relevant to the community partner’s 
experience of the community-based learning project (Charles & 
Choi, 2013, pp. 50-51). 

Recommendations From Charles and 
Choi’s Report 

As a result of their study, Charles and Choi made seven recom¬ 
mendations to the Teagle Consortium: 

• Develop separate scorecards for faculty, students and 
community partners. 

• Create a standard protocol for administering the 
Teagle Scorecard and consider using technology. 

• Refine and improve data collection and entry into, and 
management of, the Teagle Scorecard database. 

• Explicitly identify the Teagle Scorecard as a tool for 
improvement, not evaluation. 

• View the Teagle Scorecard as a tool to inform key per¬ 
formance indicators (KPIs) for effective practice in 
CBL courses. 

• Create Teagle Scorecard informed KPIs and a related 
KPI scorecard. 

• Implement the project teams research-based ongoing 
assessment model for faculty of community-based 
learning courses. 

These were among 17 recommendations in all, made to dif¬ 
ferent constituencies: institutional policy makers, faculty members 
who use community-based learning, and research funders, as well 
as to the Teagle Consortium. The last three recommendations refer 
to key performance indicators derived from the CBL Scorecard 
intended to be more easily interpretable for application to course 
practice. To our knowledge, this revised instrument has not been 
piloted. (See Charles & Choi, 2013, for further information about 
the key performance indicators and this proposed next stage.) At 
Hobart & William Smith Colleges we continue to use the most 
recent version of the CBL Scorecard, as faculty have found it helpful 
and instructive, and administering a single instrument over 6 years 
has allowed us to compare year-to-year data and analyze trends. 
Purther, we suspect that some of the faculty discontent may reflect 
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dissatisfaction with earlier formats for delivery of results. As noted 
above, during the first semesters of implementation, results were 
delivered in “raw” statistical form. The delivery format was subse¬ 
quently revised for greater readability and applicability to course 
improvement. 

Lessons Learned, Implementation Suggestions 

The development of the CBL Scorecard was an iterative effort, 
incorporating feedback from participants, with gains in face and 
content validity from the work of Braxton and Jones. Through six 
implementations of the scorecard between 2010 and 2014, the 
consortium built a database and a report format that translates 
raw scores into more readable results for faculty, organized by 
the four domains: placement quality, application and connection 
to academic learning, reflection, and quality of community part¬ 
nerships. Among the consortium members, there was agreement 
about the quality of questions and the yield of useful information. 
Yet, at the end of the grant, the loss of a central office to gather 
results from each institution, compile data, and benchmark across 
institutions, served as a deterrent to many of the consortium mem¬ 
bers. Representing an institution that found utility in outcomes 
from the CBL Scorecard, we suggest that institutions review their 
capacity to consistently implement such an instrument and their 
“comfort” with the limitation to internal assessment without cross- 
institutional comparison. 

To achieve a high response rate, we suggest the following 
approach: 

1. Initial efforts included sending the survey out elec¬ 
tronically at the conclusion of the semester, yielding 
a low response rate in some courses making it diffi¬ 
cult to interpret results. Regarding administration of 
the survey after the last day, a participating Hobart & 
William Smith Colleges professor noted, “I am sur¬ 
prised and disappointed to see that only half of my 
class responded; they were reminded, and they all 
promised. Regardless, it is very useful to see the survey 
items again, because it reinforces my course learning 
goals.” To increase the response rate, CBL Scorecard 
questions may be loaded onto SurveyMonkey or an 
institutions internal evaluation tool, and the survey 
administered during the last or penultimate class in a 
computer lab. 



Consortia! Collaboration and the Creation of an Assessment Instrument for Community-Based Learning 69 


2. Alternatively, if hosting the last class in a computer lab 
is not a feasible option, all students could access the 
survey via personal tablets or laptops or comparable 
devices on loan from Information Technology (Note: 
Although the survey is not cumbersome, the questions 
are asked in a matrix format that would lead to a frus¬ 
trating experience on personal hand-held phones.) 

3. The method we employ, after trying various 
approaches, involves distributing paper copies of the 
survey to professors who collect and return them to 
an entity (in our case, the Center for Community 
Engagement & Service-Learning at Hobart & William 
Smith Colleges) that has the capacity to enter the data 
manually 

4. It is vital to differentiate the CBL Scorecard from the 
course evaluation, since their objectives are entirely 
different, and to confuse evaluation with assessment 
of effective service-learning practices could have a 
negative impact on faculty evaluation. Administering 
course evaluations immediately before or after the 
administration of the CBL Scorecard can also generate 
“assessment fatigue” and lead to less helpful responses 
from students. 

5. Linally, we have found it effective to remind students 
during an initial service-learning training or during 
service-learning site visits that their feedback is impor¬ 
tant and that participation in the anonymous and vol¬ 
untary survey assists institutional efforts to advance 
community-based learning as a thoughtful and pro¬ 
active teaching method and a mutually beneficial col¬ 
laboration among community partners, students, and 
faculty. 

Other Challenges 

As noted, Charles and Choi (2013) found that faculty teaching 
high-performing courses commented frequently that the feedback 
from the CBL Scorecard was helpful as a guide and a reminder of 
best practices for community-based learning pedagogy. However, 
faculty concerns about the misinterpretation of the intent of the 
survey were also noted in this research. Hobart & William Smith 
Colleges addresses this concern by making every effort to assure 
faculty members that class-specific results from the CBL Scorecard 
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are shared only with the faculty member, and any aggregate data 
appears without class-identifying comments. 

All members of the consortium share the belief that commu¬ 
nity partners have a unique and vital knowledge base, and stu¬ 
dent interaction with community partners through community- 
based learning classes is a fundamental aspect of robust learning. 
When the CBL Scorecard was developed, questions incorporated 
research that accounted for this best practice. Indeed, one of the 
four domains of practice reflected in the scorecard is “quality of 
community partnerships.” For instance, in the survey for students, 
they are asked to register the degree to which statements like the 
following are true of their community-based learning experience: 
“Community partners have a clear sense of what community-based 
learning projects will accomplish for them”; “The goals of commu¬ 
nity-based learning projects carefully consider the traditions/cul¬ 
ture of the local community”; and “Community partners provide 
feedback on students’ work on the project.” However, due to chal¬ 
lenges, including community partners’ inability to take time away 
from important day-to-day operations of managing their non¬ 
profit agencies, community partners of the consortium members 
were not a part of the initial convening where the scorecard was 
developed. During subsequent consortium meetings, community 
partner voice was present and a valued component to the discus¬ 
sion. In proverbial hindsight, a better practice might have been to 
provide a stipend or comparable benefit for community partners 
who participated. 

After conversations with community members, staff at the 
Center for Community Engagement & Service-Learning at Hobart 
& William Smith Colleges willingly acknowledged the time-con¬ 
suming burden on community partners of responding to two 
evaluative tools (feedback on student performance and community 
partner scorecard) and discontinued administration of the CBL 
Scorecard to community partners. Nonetheless, we learned that 
providing feedback on student performance served as a welcome 
way for community partners to feel a part of the student learning 
process. Consequently, we request feedback through a six-question 
survey (available at http://www.hws.edu/academics/service/pdf/SL_eval- 
uation09.pdf) at the conclusion of each semester and gather com¬ 
munity partners annually in informal settings to address the ideals 
set forth in the community partner survey. Such performance feed¬ 
back is collected and provided to faculty members for inclusion in 
consideration of students’ final grades. 
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Ultimately, various consortium members employ the scorecard 
in ways that best align with their program and community engage¬ 
ment goals. The scorecard may serve as a guide to staff who support 
the faculty with opportunities to enhance the community-based 
learning experience, and to promote dialogue between community 
partners and faculty members. “As a planning document, the score- 
card informed our pedagogy and ultimate approach. Essentially, 
it served to remind us that community partner involvement and 
reliable, regular contact was essential,” reported Jay Szczepanski, 
director of the Learning Resource Center at Flagler College (per¬ 
sonal communication, December 15, 2013). Other institutions found it 
useful for faculty development opportunities around community- 
based learning, as a formative rather than summative tool. As Lisa 
Wolfe from Franklin & Marshall College’s Ware Institute for Civic 
Engagement commented: 

In the spirit of collegiality, we have used the scorecard 
as an educative tool. Especially for new practitioners, it 
clearly summarizes the standards for high-quality/high 
impact community-based learning and can be used as 
a rubric for identifying areas where there is the most 
potential for growth, (personal communication, March 13, 

2015) 

At Hobart & William Smith Colleges, we have used CBL 
Scorecard results to guide planning of faculty enrichment opportu¬ 
nities and the scorecard itself for discussion with faculty of effective 
community-based learning pedagogy. Professor Mary Kelly from 
our Education Department noted: 

The scorecard provides us with a reflective tool to assess 
the degree to which our service learning projects uti¬ 
lized best practices. It was a good reminder that, for 
example, participating in a service learning project 
along with students could have a positive impact on 
learning outcomes, and can be a lot of fun, too. (personal 
communication, July 18, 2016) 

Conclusion 

To return to our initial research goals and questions, to what 
degree has the CBL Scorecard met the Teagle Consortium aims 
of assessing community-based learning with an instrument that 
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employs a quantitative scale to afford benchmarking across institu¬ 
tions and provides a mechanism for closing the assessment loop, 
offering feedback to faculty to improve student learning? A recent 
evaluation of the research assessing the cognitive outcomes of com¬ 
munity-based learning concludes: “Overall, these empirical inves¬ 
tigations of cognition in service learning and related pedagogies 
support the importance of well-integrated service learning, the role 
of critical reflection, and the value of assessment methods that are 
grounded in theory” (Fitch, Steinke, & Hudson, 2013, p. 68). All three 
of these goals are central to the CBL Scorecard and its aims and 
development: CBL Scorecard questions draw on widely recognized 
research to promote the most effective pedagogical practice for stu¬ 
dent learning, and among the hallmarks of that practice are the full 
integration of the experience in the community with course con¬ 
tent and reflection activities that are relevant and encourage critical 
thinking. The “success factors” promoted by the CBL Scorecard 
are “environment variables that correlate in the research with such 
standard student learning and developmental outcome measures 
as GPA, writing skills, critical thinking skills, leadership, values, 
career choice and post-college activity” (Rhodes College, Assessment 
of Student Learning, 2008, p. 8). As a result, we are confident that this 
instrument helps further the aims of a liberal education through 
the enhanced cognitive and affective development of our students, 
gained in robust and meaningful community-based learning expe¬ 
riences. In other words, strengthening the effectiveness of commu¬ 
nity-based learning courses and programs through ongoing assess¬ 
ment bolsters learning in the liberal arts, especially the integrative, 
collaborative, interdisciplinary, and multidisciplinary learning that 
will best prepare students for a rapidly changing, increasingly com¬ 
plex and interdependent world. 

Although hardly a “silver bullet,” this instrument also addresses 
some of the challenges facing the assessment movement, most 
specifically the issue of “closing the loop.” A major concern has 
been how to move from assessment of learning to improvement 
of learning based on assessment results. As Blaich and Wise (2011) 
remark: 

Although much of the national conversation about 
assessment and accountability focuses on the pros 
and cons of different approaches to measuring stu¬ 
dent learning and experience, we have learned from 
the Wabash Study that measuring student learning and 
experience is by far the easiest step in the assessment 
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process. The real challenge begins once faculty, staff, 
administrators, and students at institutions try to use 
the evidence to improve student learning, (p. 3) 

Ewell (2009) sees a lack of specifics in assessment evidence as 
part of the difficulty in moving from assessment to improvement: 

Many institutions simply do not know, however, how to 
implement evidence-based continuous improvement. 

Why is this? One reason is the general nature of most 
assessment results when compared to the concrete reali¬ 
ties of changing curriculum and pedagogy. Although 
exceptions are apparent, most assessment evidence 
is simply not fine grained enough to yield actionable 
information at this level. A similar reason is that such 
evidence tends to be presented in the form of central 
tendency measures, which don’t show the patterns of 
strength and weakness or the variations in performance 
across types of students needed to guide intervention 
(Kuh, 2007). Finally, information about outcomes alone 
doesn’t tell faculty what to fix. (p. 16) 

Admittedly, the CBL Scorecard does not yield direct evidence 
of student learning, so its results may be less easily interpretable 
into changes in classroom practice. However, it does give faculty 
feedback with clear and direct application to the community-based 
course they teach. That the results are course-specific is one of its 
strengths. In short, its summative role feeds into a formative one. 
As one faculty member put it: “It helped me realize what I wasn’t 
doing that I needed to do” (Charles & Choi, 2013, p. 49). 

With the end of the Teagle grants, we no longer had funding 
for a central mechanism through which to gather responses from 
various institutions and produce cross-institutional benchmarking. 
Nonetheless, the instrument has the potential for benchmarking, 
as our pilot demonstrated. At Hobart & William Smith Colleges we 
plan continued use of the CBL Scorecard, with results from each of 
the past 10 semesters building internal benchmarking capacity to 
guide adjustments to individual courses and programwide altera¬ 
tions. Such capacity provides individual faculty with the data to 
understand anomalies that may occur and thereby to strengthen 
the effectiveness of service-learning in their courses over time. 
This benchmarking capacity also provides us with aggregate data 
to review with our Service-Learning Advisory Council in a broader 
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effort to promote institutional conversations about how to ensure 
high-quality service-learning experiences and enhanced student 
learning outcomes. Research is under way that would use scorecard 
results to further illuminate the strengths and weaknesses of CBL 
offerings, such as their impact on different student cohorts and 
related questions. In addition, from our perspective as an insti¬ 
tution with the Carnegie community-engaged classification, we 
find the CBL Scorecard helpful in documenting and encouraging 
the practices represented by that certification. Ideally, there would 
be an interinstitutional mechanism to compile data and promote 
cross-institutional benchmarking, but until that opportunity arises, 
we will proceed at current capacity. 

As a last note, we must add that in addition to the individual 
institutional uses of the CBL Scorecard, there is another compelling 
consequence of this collaborative effort: the formation of a com¬ 
munity of practice where colleagues have established relationships 
that support personal development and institutional growth. These 
are by-products that will indeed be enduring. 
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